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Abstract 

Robert Burns wrote: “ The best laid schemes of Mice and Men oft go awry ”. This could be 
considered the motto of most educational innovation. The question that arises is not so much why some 
innovations fail (although this is very important question), but rather why other innovations succeed? This 
study investigated the success factors of large-scale educational innovation projects in Dutch higher 
education. The research team attempted to identify success factors that might be relevant to educational 
innovation projects. The research design was largely qualitative, with a guided interview as the primary 
means of data collection, followed by data analysis and a correlation of findings with the success factors 
identified in the literature review. In order to pursue the research goal, a literature review of success 
factors was first conducted to identify existing knowledge in this area, followed by a detailed study of the 
educational innovation projects that have been funded by SURF Education. To obtain a list of potential 
success factors, existing project documentation and evaluations were reviewed and the project chairs and 
other important players were interviewed. Reports and evaluations by the projects themselves were 
reviewed to extract commonalities and differences in the factors that the projects felt were influential in 
their success of educational innovation. 

In the next phase of the project experts in the field of project management, project chairs of 
successful projects and evaluators/raters of projects will be asked to pinpoint factors of importance that 
were facilitative or detrimental to the outcome of their projects and implementation of the innovations. 

After completing the interviews all potential success factors will be recorded and clustered using an affinity 
technique. The clusters will then be labeled and clustered, creating a hierarchy of potential success factors. 
The project chairs will finally be asked to select the five most important success factors out of the 
hierarchy, and to rank their importance. This technique - the Experts' Concept Mapping Method- is based 
upon Trochim ’s concept ma pping approach ( 1 989a, 1 989b ) and was developed and perfected by Stoyanov 
and Kirschner ( 2004 ). 

Finally, the results will lead to a number of instruments as well as a functioned procedure for 
tendering, selecting and monitoring innovative educational projects. The identification of success factors 
for educational innovation projects and measuring performance of projects based upon these factors are 
important as they can aid the development and implementation of innovation projects by explicating and 
making visible (and thus manageable) those success and failure factors relating to educational innovation 
projects in higher education. 

Determinants for Failure and Success of Innovation Projects: The Road to 
Sustainable Educational Innovation 

The Dutch Government has invested heavily in stimulating better and more creative use of 
information and communication technologies (ICT) in all forms of education. The ultimate goal of this 
investment is to ensure that students and teachers are equipped with the skills and knowledge required for 
success in the new knowledge-based economy. All stakeholders (i.e., government, industry, educational 
institutions, society in general) have placed high priority on achieving this goal. However, these highly 
funded projects have often resulted in either short-lived or local successes or outright failures (see De Bie, 



359 




2003; Teasly, 1996). As a result, the role of ICT is developing less quickly in higher education institutions 
than was previously expected. 

In order to steer these developments in the right direction, SURF, a government-funded national 
organization in which all higher education institutes in the Netherlands participate to increase the pace of 
educational innovation, first set up the SURF Educatie<F> (SURF Education) program for educational 
innovation projects and has followed this up with the SURF ICT and Education Platform. The goal of this 
platform is the systematic stimulation of the application of ICT to innovation in higher education via: (1) 
professionalization of personnel, (2) effective deployment of resources, (2) monitoring the durability of 
(digital) educational material, (3) facilitating measurable improvements in education as a result of the 
deployment of ICT, (4) fostering a systematic approach to innovation and the development and 
dissemination of knowledge, and (5) promoting the use of standards. Finally, SURF’s educational 
innovation projects are intended to be a source of inspiration for the introduction of ICT-based innovation 
in education and are aimed at better and sustainable results. It’s educational innovation projects often cover 
one or more of the following key issues: 

• Competence/Portfolio 

• Collaborative learning 

• Interactive teaching materials 

• Learning content management systems (LCMSj/Communities 

• New media 

This is all very noble, but after having funded projects for five years, SURF asked itself three 
important questions with respect to educational innovation, namely: Why are some innovations more 
successful than others? Why do some innovations fail, while others succeed? and How can an innovation 
be sustained once the grant funding has ended? The present project, funded by the SURF ICT and 
Education Platform, tries to answer these questions by identifying the determinants for success and failure 
of large-scale educational innovation projects in Dutch higher education, in particular of those funded by 
the SURF ICT and Education Platform. The identification of success and failure factors for educational 
innovation projects and measuring the chances and performance of projects based upon these factors are 
considered important to SURF Education as they can aid the development and implementation of 
innovation projects by explicating and making visible those success and failure factors relating to 
educational innovation projects in higher education. This is necessary to make these factors manageable 
and to enable future projects to achieve better and more sustainable results. 

The research project consists of the following phases: 

• A literature study to determine the benchmarks for success and failure of educational innovation 
projects. 

• Analyses of the SURF projects started in 1999, 2000, and 2001 on the basis of the benchmarks from 
the literature study and the identification of new benchmarks from these projects. 

• Interviews with experts from SURF Education, project managers of the SURF Education projects and 
experienced (commercial and governmental) project managers and advisors to determine the subjective 
dimension of success and failure. 

• Development of a number of tools for evaluating project plans and progress as well as a new procedure 
for tendering for funds. 

• Dissemination of the results by writing articles for professional and scientific journals and the 
presentation of the results at national and international conferences and workshops. 

This contribution presents the results of the first phase of the research project; the literature review 
of success factors of educational innovation projects to identify the existing knowledge in this area. 

A first step in creating a meaningful report is finding an answer to the questions of how ‘success’ 
can be operationally defined and how the dimension ‘success of educational innovation’ can be tapped. A 
problem here is that there are two types of ‘success’, namely success of the project and success of the 
innovation. These two ‘successes’ are completely different. Neither is a requirement nor a guarantee for the 
other. In general, success can be seen as the accomplishment of goals and objectives necessary to achieve a 
particular task. SURF ICT and Education Platform projects are considered successful based on the extent to 
which they stimulated and facilitated new and better use of ICT in education. SURF is particularly 
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interested in the sustainability of the technological innovations, in other words, how they can ensure that 
the innovations, both technical and pedagogical, make the shift from an externally funded initiative to a 
sustained ‘standard operating procedure’. The rationale for sustaining successful ICT -based educational 
innovations is to preserve what has been valued and built for continued use. 

Inherent to the concept of success factors is a notion that if success factors are implemented / 
heeded in the project, the educational innovation or project will perform better and be successful. Success 
factors can thus be looked upon as individual independent variables influencing the dependent variable 
‘success’. Analogously for failure factors, there is a notion that if failure factors are avoided in the project 
or implementation, that the innovation will have a better chance of being achieved. Performance 
measurements for success or failure can be classified as objective and subjective. The objective measures 
are based on measurement of past performance or output while subjective measures let individuals with an 
intimate knowledge of educational innovation be the judges of what is successful. It should be recognized 
that success of an innovation at the project level does not automatically generalize to success at the 
institutional or national level, but needs careful considerations of scalability, generalizability, temporal 
flexibility, and financial sustainability. To be able to determine which of the identified success factors for 
educational innovation projects are most relevant in different educational contexts, a contextual framework 
to position educational innovation and related success factors needs to be developed. The basic idea is that 
if we know why some projects fail or have only short-lived successes and we can avoid making these 
mistakes, it is possible to make sure that new projects will succeed with sustainable results. However, it 
should be noted that ‘non-failure’ is not considered the same as success. Success/failure should not be 
considered as a binary classification, but should be viewed along a continuum. 

Although, SURF sets clear goals for applicants to address sustainability in their project plan, and 
generally, clear implementation plans are available, clear plans for sustaining the innovation are lacking. 
Sustainability seems to be more an afterthought rather than a planned strategy for maintaining change. 
Kenny and Meadowcroft (1999) suggest that forward thinking and vision are paramount in successfully 
planning sustainable developments. The observations presented in the literature study are meant to 
contribute to the understanding of why and how technological innovations in education are adopted and 
diffused. Together with the outcomes of the other phases of this project, the report is aimed at supporting 
SURF to steer the innovation projects towards success and sustainable technological innovations and to 
create mechanisms that empower all stakeholders to sustain innovative developments. 

For this literature study, we looked for books, accessed ERIC®, PsycINFO®, and used Google® 
to search the World Wide Web using the following search terms: success, success factors, failure factors, 
success determinants, innovative projects, innovation, return -on-investment, project, innovative factor, 
educational innovation, critical success factor, criteria, guidelines, project management, sustainability, 
success conditions, innovation processes, PT3, standards, Europe, strategy. We located over eighty articles 
specifically addressing the topic of interest. Reading these and following up cross-references we 
established a knowledge base of over thirty books, articles, websites, and papers that addressed different 
aspects of why educational innovations sometimes fail and sometimes succeed. Also, it has resulted in the 
following structure for the rest of the review. 

This report has the following structure. Since innovation can be considered a design and 
implementation exercise composed of a number of distinct phases, the different phases that can be 
distinguished in innovative projects are first described. Second, the business or corporate view on the 
success of innovative projects is described. In business there is a long tradition of using (technological) 
innovation as a major instrument to compete, survive and grow. In comparison, the field of education has 
only recently started to worry about competition and surviving. Third, the experiences from educational 
projects are considered to determine the success and failure factors of innovation projects. Fourth, the 
factors that determine the sustainability of innovations are discussed. Finally, the different experiences are 
synthesized in an integrative model that can be used by SURF to assess project proposals to SURF 
Education regarding their potential to accomp lish sustained innovations, to recognize project warning 
signs, and to increase the chance of success of the diffusion and permanence of the innovations. 

Literature review 

Project phasing 

Innovating and changing an organization is becoming more and more complex because 
organizations and the relationships between organizations are becoming more complex. Factors influencing 
this complexity are level and type of technology, environmental influences, size and structure of the 
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organization (e.g., tendency towards fusions), interdependence between organizations (e.g., tendency 
towards IT production), willingness to change (e.g., overcoming human and organizational inertia), lack of 
support from the management, time and money constraints, and so on. Innovating (or changing) the 
structure of an organization often comes up against a wall of resistance. Not surprisingly, the human factor 
is often considered the most influential factor on the chance of success. Innovating an organization or 
structure places a heavy burden on the organization and the employees and therefore it is necessary to have 
insight in the complex matter of the phases of organizational changes, the way people deal with innovation 
processes, the methods and strategies for change. 

Katz and Kath ( 1 978), for example, argue that planning to initiate an innovation can be done in at 
least three phases: a diagnostic phase in which aspects like how the organization is structured and 
organized need to be addressed, a goal-setting phase where the goals need to be set and the design needs to 
be specified and an innovative-process phase consisting of determining which strategies, roles, methods 
and interventions can best be used to realize the desired situation. Kor and Wijnen (2001) specify this 
concept of phasing a bit further and according to them a project can be divided into six phases: initiation, 
definition, design, preparation, realization and maintenance. Another more condensed and very useful 
model in which the maintenance (or sustainability) of the innovation is explicitly mentioned, is presented 
by Alexander and and McKenzie (1998) and Fullan (1991). In their view on the phasing of organizational 
innovation and change, they distinguish between three main types of phases: an initiation phase in which 
planning and evaluation takes place, an implementation phase in which the development, implementation 
and evaluation of the project takes place, and finally a maintenance or institutionalization phase regarding 
the sustainability of the project. We have opanded this model as shown in Figure 1. 
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Figure 1 Phasing 

Corporate / business point of view 

Within a corporate or business point of view, research on successful projects has been done by 
different studies. Some striking results come from the Standish Group 2000 Study (Johnson, Boucher, 
Connors, & Robinson, 2001): Only 28 % of the IT projects investigated were successful. Since not all un- 
successful projects can be rated as failed the Standish Group 2000 study categorises projects into three 
types: 

• Successful (28%): project is completed on time and on budget with all features and functions originally 
specified; 
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• Challenged (49%): project is completed and operational, but over budget, late, and with fewer features 
and functions than initially specified; 

• Failed (23%): project is cancelled before completion, or never implemented. 

The Standish Group 2000 study and numerous other studies conclude that the human factor is 
often considered as the most influential factor on the chances of failure or success (e.g. Johnson et al., 

2001; Kor & Wijnen, 2001; Schein, 1995; Storm & Jansen, 2004; Turner, 1999) 

Educational point of view 

In addition to the above mentioned studies which were conducted from a corporate or business 
point of view, there has been a good deal of research within educational settings into the determinants of 
success or failure of innovative projects. A two-year national study to evaluate the contribution of 
information and communication technology projects to student learning in higher education was conducted 
in Australia by Alexander and McKenzie ( 1998). The study reviewed over 100 projects, which received 
teaching developments grants and made significant use of a range of information and communication 
technologies to develop student learning materials. After a detailed literature review, a questionnaire was 
developed and sent to the project leaders of 173 projects across Australia and finally these questionnaires 
were analyzed. A striking outcome of the questionnaire was the large discrepancy between the intended 
outcomes of the projects and the actual outcomes reported. While 87% of the projects’ leaders noted 
“improved quality of learning” to be an intended outcome of the project, only 30% reported this as an 
actual outcome . Although this discrepancy can be due to failure of the project and/or failure of the project 
to measure this, it seems justified to state that the majority of the projects have not been successful in 
achieving their intended outcome. Furthermore, the study has shown that technology on itself, does not 
make a difference improving learning outcomes or assures successful educational innovations. 

Diffusion is the process by which an innovation is communicated through certain channels over 
time among members of a social system. The diffusion of innovations theory (Rogers, 1995) helps to 
explain and analyze how an idea or proposal for an innovative project is communicated and accepted by 
others. Therefore, this theory affects the initiation phase and needs to be taken into account when initiating 
an innovative project. According to Rogers (1995, p. 6), “diffusion is a kind of social change, defined as the 
process by which alteration occurs in the structure and function of a social system.” Rogers also defined 
adoption as being a decision to make full use of an innovation and rejection as a decision not to use an 
innovation. It can be concluded that innovations that are perceived by potential adopters as having greater 
relative advantage, compatibility, testability, observability and less complexity will be adopted more easily 
than other innovations. 

Educational innovations, such as implementing a new pedagogic method such as problem-based 
learning, are sometimes the result of an intuitive and hasty decision to change. And without a proper 
investigation of aspects as problems the innovation is supposed to solve, needs of the future users, 
willingness of different participants to cooperate and success and fail factors of this particular project, the 
success of the innovation is at stake. According to De Bie (2003), in order to be successful, projects need to 
take nine success factors into account when initiating an innovation: vision, strategy of development, 
acknowledgement of problems, project plan, project management, resources, role of the “outside world”, 
support from the rest of the institute and competent management. 

Apart from these process aspects, educational innovation has another important aspect, namely the 
content. Both process and content need to be taken into account in order for the project to be successful. 

The content aspect of the educational innovations not only concerns the actual content of the innovation as 
seen in the goals of the project plan, but, according to Vinkenburg (2003), also includes the knowledge, 
skills and attitudes of the people involved in the innovation -as far as this concerns new and to be 
innovated knowledge, skills and attitudes in order to let the innovation succeed. Process aspects are highly 
dependent on the competence of the management, a human factor. Numerous projects fail due to 
incompetent leadership or management (e.g., Alexander & McKenzie, 1998; De Bie, 2003; De Koning & 
Florijn, 1995; Holmes, 2001; Johnson et al., 2001; Kor & Wijnen, 2001; Mathias & Rutherford, 1983; 
Rutherford, 1992; Storm & Jansen, 2004), lack of support from the head of department, dean or other 
person in authority (e.g., Alexander & McKenzie, 1998; Hannan, English, & Silver, 1999; Johnson et al., 
2001; Light, 1998) or lack of support from the rest of the organisation and peers (e.g,. Hannan, English & 
Silver, 1999; Light, 1998; Zhao, Pugh, Sheldon, & Byers, 2002). 
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Sustainability 

As mentioned previously, an innovation process takes place in three stages: initiation, 
implementation and the institutionalization. Although the last phase is often forgotten or neglected (due to 
lack of money, time, etcetera), it is very important. In order to make the time, money and effort worthwhile 
it is important to carefully plan this last phase. The implementation phase may continue for a period of 
time, but eventually there should be a point at which the new idea becomes institutionalized and regularized 
as a part of the ongoing operations. It is now no longer an innovation process, but rather a normal process. 
And whether or not the innovation becomes a durable part of the organization depends on the commitment 
and action of the participants as well as on other factors. 

Schein (1995) found that human change, whether at an individual or group level, is a 
psychological process that involves painful unlearning an relearning while individuals attempted to 
restructure their thoughts, perceptions, feelings and attitudes. People need to un-freeze, change and re- 
freeze. Un-freezing refers to the removal of the restraining or balancing loops that are often associated with 
group norms and leads to cognitive dissonance or conflict that can be very disorienting to group members 
as they begin to change. When dealing with such a disorientation or disequilibrium, the group members 
need to change or reframe their thought process, their ideas and representations of what is “normal’' and 
interpret new concepts more broadly than before. This is called re-freezing. The key to effective change (or 
innovation) is to carefully dose the amo unt of change, and therefore of the perceived threat, produced by 
the disconfirming information to allow the group members to feel safe to un-freeze, change and re -freeze 
their ideas and concepts. But just to receive disconfirming information is not enough to change. To become 
motivated to change, you must accept the information and connect it to something you care about. The 
information must be valid and relevant. Furthermore, O’Hara, Watson, and Kavan (1999) state that the 
more an innovation plans to change, the greater the influence from the environmental will become, the 
greater the risks will be and the chances of failure will increase. 

Light (1998) noticed that many innovations are deserted not at the initiation or implementation 
phase but in the institutionalization phase. He further argued that four main factors influence the degree to 
which innovations are sustained through this institutionalization phase: external environment, internal 
structure, leadership and internal management. Light suggests with this model that when changing the 
structures and culture of an organization, you need to turn the traditional organization into a “learning 
organization’’, that is an organization that adapts to innovations and restructures itself to accommodate 
change. 



Synthesis 

In this paragraph we synthesize the different (educational and business) perspectives and present 
an integrative model that might be used by SURF as a guide in establishing policies and procedures, in 
formulating new guidelines for project proposals, in the assessment of SURF ICT and Education Platform 
innovation project proposals, in conducting a project’s health check to identify warning signs for failure, 
and in promoting the diffusion and sustainability of the projects’ innovations. The model presented below 
shows the factors that are considered imperative to accomplishing a successful project, categorized by 
project stage. These factors are formulated in such a way that they can easily be converted into a checklist 
format that can be used by SURF in the assessment of the project (proposals and reports) and by applicants 
in the writing of project proposals. It should be noted that although these variables are identified as major 
contributors to project success, they will never guarantee success alone. 

Initiation : 

In the initiation phase the present situation needs to be assessed in terms of goals, problems, and 
discrepancies. Then, ideas for the new situation need to be planned taking into consideration the changes 
envisioned and the route from the present to the new situation. Next, the consequences of the effects of the 
innovation on other aspects of the organization need to be evaluated. The following factors are considered 
important in this phase: 

• There should be clear project objectives 

• The general mission of the project should be clearly defined 

• The project’s scope should be adjusted in such a way that the level of changes needed on the route 
from the present to the new situation can be clearly envisioned 

• (Simplified) Return -On-Investment should be taken into account 
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• Stakeholders should be identified 

• Formal feedback channels must be created 

• There should be an experienced project manager 

• The project manager should not be the organizational manager 

• The project manager should be given responsibility and authority 

• The project team members should be competent 

• Care must be taken of clear responsibility and accountability of team members 

• The project manager and team members should be able to explain their efforts and results in ways that 
the larger organization can understand 

• Project manager (team) should be prepared to re -plan 

• The project manager and team members should listen to resistors of innovation because they are often 
aware of unintended consequences of the innovation 

• There should be commitment from executive management and peers 

• There should be support from executive management and peers 

• Realistic expectations should be created 

Implementation: 

In the implementation phase the innovation is developed, implemented, and evaluated. The 
following factors are considered important in this phase: 

• User involvement should be ensured 

• User expectations about the innovation should be managed 

• Stakeholders should be engaged 

• Initiator of the change should be trusted and respected by the prospected users 

• Adequate communication channels should be created 

• Focus should be on adoption rate of approximately 25% of the system members 

• Focus should be on affecting opinion leaders’ attitudes (the more opinion leaders adopt the system, the 
lower the critical adoption rate for other system members will be) 

• Project manager and team should be open to external criticism 

• Project manager and team should continually question own assumptions 

• Project manager should continue to modify plan based on realities 

Institutionalization: 

Institutionalization means continuing the newly implemented change or stabilizing the use of an 
innovation (Sherry, 2003). In the institutionalization phase the innovation needs to be scaled and 
maintained. The following factors are considered important in this phase: 

• The organization should be fitted in a stable environment which is supporting and collaborating 

• The organization should have a relatively loose, centralized structure with good vertical 
communication channels 

• There should be competent leadership and management 

• The amount of change should be carefully dosed 

• User expectations should be managed: Innovation projects will fail if the users of a system are 
dissatisfied with it because it does not meet their expectations. Therefore, project managers should not 
only manage the development of the system, but also the perception of the system. 

• The innovation’s relative advantage as compared with the current practices should be communicated 

• Structures should be created that promote learning of new practices and observable incentive systems 
that support them 

• Effective communication among all parts of the system should be created 

• A high degree of observability, that is, a degree to which other persons than the innovator see its 
results as beneficial, should be created 

Since the educational innovations that are sponsored in the SURF ICT and Education Platform 
program take place within the time and funding limitations of SURF’s grant, and the sustainability of the 
innovations is considered a major goal by SURF, it seems necessary to require applicants to present a 
detailed plan for sustaining the innovation and to challenge them to realize this plan after the project has 
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ended, i.e. once the funding period of the innovation grant is over. To accomplish this, the success factors 
that are imperative for sustainable innovations to materialize, must be clearly communicated to the 
applicants/project managers. This can be done by using the Project Health Checklist. This Project Health 
Checklist (PHC-list) can be used during the different phases of the project by the project manager and team 
members in order to check the ‘health’ and progress of their project (i.e., are all important determinants for 
success taken into account in this project or do we need to adjust the project?). In addition, project 
managers and team members need to be familiarized with methodologies that can be used in the 
institutionalization phase to attain sustainable innovations. In this respect, CATWOE a Soft Systems 
Methodology initiated by Checkland and Coles (1990) seem promising. Furthermore, we believe that a 
financial incentive could be used to challenge the project team and the responsible educational institutions 
to maintain the innovation and diffuse it to other institutes. Currently, there is no financial incentive to 
sustain innovations. Lack of funding in this phase is certainly a disincentive, especially when adopting an 
innovation means that individuals must go through a learning curve and take on new responsibilities as a 
result of developing expertise (Sherry, 2003). Alternatively, it would be possible to make part of the 
funding conditional to the realization of the plan for sustaining the innovation. Regardless of which 
solution is chosen to promote the sustainability of an innovation, there must be a means to protect the 
organization if the innovation proves too costly to sustain. 

The identification of success and failure factors of educational innovation projects by means of 
literature study was the main goal of the first stage of this research project. In the second stage these 
‘objective’ factors will be used to analyze the SURF Education projects that have started in 1999, 2000, and 
2001 (see appendix I). More specifically, the plans and reports of these projects will be studied to 
determine if and how they have taken account of these success and failure factors, and to see if additional 
success and failure factors can be identified in these projects. In the third stage, the main people involved in 
the SURF Education projects will be interviewed to extend the ‘objective’ dimension of success and failure 
with a subjective dimension. It is expected that the overall results that emerge from these three stages can 
be used by SURF in their pursuit of successful projects with sustainable educational innovations. 

Although, this literature review has shown that the predominantly retrospective analyses of 
innovation projects can reveal interesting models, comprising numerous failure and success factors of 
innovation projects, there seems to be no firm empirical basis for these models. Therefore, it seems 
necessary to conduct more systematic research into the mechanisms that cause project success or failure. 

In the next phase of the project experts in the field of project management, project chairs of 
successful projects and evaluators/raters of projects will pinpoint factors of importance facilitative or 
detrimental to project outcomes and innovation implementation. After the interviews all potential success 
factors will be recorded and clustered using an affinity technique. The clusters will be labeled and 
clustered, creating a hierarchy of potential success factors. The project chairs will finally select the five 
most important success factors out of the hierarchy, and to rank their importance. This technique - the 
Experts’ Concept Mapping Method- is based upon Trochim’s concept mapping approach (1989a, 1989b) 
and was developed and perfected by Stoyanov and Kirschner (2004). 
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